Skip to content

ci(testnet): deterministic reproduction of the bootstrap livelock, debug logs on chosen nodes - #78

Open
EnzinoBB wants to merge 3 commits into
masterfrom
ci/bootstrap-pause
Open

EnzinoBB wants to merge 3 commits into
masterfrom
ci/bootstrap-pause

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Fable analysed the bootstrap-livelock stall that still happened with #77 (run 39886642406). The stage-1 hash race was not the cause. The real cause is timing, and it is official behaviour:

  • the starter re-sends its bootstrap table every round_elapse_time;
  • every same-round table restarts the consensus on each node;
  • a round with one late stage message needs the fallbacks, which take 12.5 to 16.5 s;
  • so with a 10 s timer the round never completes.

Master shows the same stall (run 39866725719).

Changes

  • PAUSE_AT / PAUSE_SECONDS: a node that is trusted in the current round (not n1) is stopped with SIGSTOP and then continued. Every run gets a bootstrap with a silent member.
  • LOG_DEBUG_NODES: those nodes log at debug level, so stage and timer messages are visible.
  • New scenario bootstrap-pause: 10 s timer, a 15 s pause, debug logs on n1 to n3. It runs only when named, and it is expected to fail until the node stops restarting the consensus on a re-sent table.
  • Comments: the comments on bootstrap-livelock and ROUND_ELAPSE_MS now name the real cause.

Result: the pause does not reproduce the livelock

bootstrap-pause was run twice on master (runs 39896667779 and 39896678331). Both passed: 1,500 blocks with matching digests.

  • Run 39896667779 paused n2 at 634; the network kept going without it (it was at 642 when n2 continued).
  • Run 39896678331 paused n2 at 603; the network got past the bootstrap within the 15 s pause (it was at 604 when n2 continued).

So a silent trusted node with a 10 s timer is not enough on its own. The livelock seen under loss needs more, probably continuous message delays on two confidants. The debug logs of these runs (logs-bootstrap-pause, n1 to n3) are the starting point.

Keep this PR for the tools (PAUSE_AT, LOG_DEBUG_NODES) and the corrected comments. bootstrap-pause stays a manual scenario, and its comment should not claim that it fails.

🤖 Generated with Claude Code

EnzinoBB and others added 3 commits October 23, 2026 05:44
…; debug logs on chosen nodes

Fable's analysis of the bootstrap-livelock stall with #77 (run 39886642406): the cause is not the
stage-1 hash race but timing. The starter re-sends its bootstrap table every round_elapse_time and
every same-round table restarts the consensus on each node; a round with one late stage message
needs the fallbacks (12.5-16.5 s), so with 10 s it never completes. Official behaviour (master
showed the same stall, run 39866725719).

- PAUSE_AT / PAUSE_SECONDS: a trusted node (not n1) is stopped with SIGSTOP and continued, so a
  bootstrap with a silent member happens every run
- LOG_DEBUG_NODES: those nodes log at debug level (stage and timer messages)
- robustness: bootstrap-pause (10 s, 15 s pause, debug on n1-n3), run by name, fails until the node
  stops restarting the consensus on a re-sent table; comments name the real cause

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a reproduction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery node

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant